Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns

Περιέχει το πλήρες κείμενοThis paper discusses the evolution of large-scale evaluation campaigns and the corresponding evaluation infrastructures needed to carry them out. We present the next challenges for these initiatives and show how digital library systems can play a relevant role in supporting the research conducted in these fora by acting as virtual research environments

Geographic information retrieval in a mobile environment: evaluating the needs of mobile individuals

This paper describes research that aims to define the information needs of mobile individuals, to implement a mobile information system that can satisfy those needs, and finally to evaluate the performance of that system with end-users. First a review of the emerging discipline of geographic information retrieval (GIR) is presented as background to the more specific issue of mobile information retrieval. Following this, a user needs study is described evaluating the requirements of potential users of a mobile information system; the study finds that there is a strong geographic component to users' information needs. Next, four geographic post-query filters are described which attempt to represent the region of space associated with an individual's query made at some specific spatial location. These filters are spatial proximity (distance in space), temporal proximity (travel time), speed-heading prediction surfaces (likelihood of visiting locations) and visibility (locations that can be seen). Two of these filters — spatial proximity and speed-heading prediction surfaces — are implemented in a mobile information system and subsequently evaluated with users in an outdoor setting. The results of evaluation suggest that retrieved information to which post-query geographic filters have been applied is considered more relevant than unfiltered information, and that users find information sorted by spatial proximity to be more relevant than that sorted by a prediction surface of likely future locations. The paper closes with a discussion of the wider implications of these results for developers of mobile information systems and location-based services

The controlled versus natural indexing languages debate revisited: a perspective on information retrieval practice and research

This article revisits the debate concerning con trolled and natural indexing languages, as used in searching the databases of the online hosts, in-house information retrieval systems, online public access catalogues and databases stored on CD-ROM. The debate was first formu lated in the early days of information retrieval more than a century ago but, despite significant advances in technology, remains unresolved. The article divides the history of the debate into four eras. Era one was characterised by the introduction of controlled vocabulary. Era two focused on comparisons between different indexing languages in order to assess which was best. Era three saw a number of case studies of limited generalisability and a general recognition that the best search performance can be achieved by the parallel use of the two types of indexing languages. The emphasis in Era four has been on the development of end-user-based systems, including online public access catalogues and databases on CD-ROM. Recent developments in the use of expert systems techniques to support the representation of meaning may lead to systems which offer significant support to the user in end-user searching. In the meantime, however, information retrieval in practice involves a mixture of natural and controlled indexing languages used to search a wide variety of different kinds of databases